Content reproduction device, picture data output device, content creation device, content reproduction method, picture data output method, and content creation method

ABSTRACT

Original pictures photographed from a plurality of photograph viewpoints are prepared, display pictures for regions corresponding to visual-lines are generated while switching to an original picture photographed from another viewpoint is being performed according to movement of a subject  81  or the like. The original pictures are associated with tag data indicating the positions of images of particular subjects. When the viewpoint is switched, the visual line is controlled such that the position indicated by a tag for a same subject is included within the visual field before and after the switching. Accordingly, a post-switching region is displayed.

TECHNICAL FIELD

The present invention relates to a content reproduction device thatreproduces content involving picture display, a content creation devicethat creates content, a picture data output device that outputs picturedata for use in content, and a content reproduction method, a picturedata output method, and a content creation method to be respectivelyperformed by the devices.

BACKGROUND ART

Cameras capable of taking whole-sky pictures or wide-angle picturesextremely close to whole-sky pictures by means of fisheye lenses or thelike, are becoming familiar. If a picture taken by such a camera isviewable as a display target from a free viewpoint through a headmounted display or a cursor operation, the picture world can be enjoyedwhile giving a high immersion feeling, or states in various places canbe presented.

[Summary] [Technical Problem]

When an angle of view of a picture for use in display is widened, thepicture can be more dynamically displayed, but some inconveniences arecaused due to a significant increase in the degree of freedom of aviewpoint or visual line when the display region is defined. Forexample, in a case where a viewer views a wide angle picture whilechanging the viewpoint or visual line, a target that the vieweroriginally wanted to watch or that the viewer watched a while ago may bemissed so that a great effort is required to display the target again.In addition, in a case where pictures taken from a plurality ofviewpoints are switchingly used for display, if display after theswitching is quite different from that before the switching, the viewermay get confused.

The present invention has been made in view of the above problems, andan object thereof is to provide a technology of continuing picturedisplay in a preferable visual field by using a whole-sky picture.

Solution to Problem

One aspect of the present invention relates to a picture data contentreproduction device. The content reproduction device includes a dataacquisition section that acquires data on an original picture for use indisplay, and tag data indicating a position of a particular subject inthe original picture, a display-viewpoint acquisition section thatacquires a display-time viewpoint and a display-time visual line withrespect to the original picture, a display-picture generation sectionthat generates, from the original picture, a display picture in a visualfield based on the display-time viewpoint and the display-time visualline, and a data output section that outputs data on the display pictureto a display device, in which, in a state where a predeterminedcondition is satisfied, the display-picture generation section changesdisplay on the basis of the position in the original picture indicatedby the tag, by referring to the tag data.

Another aspect of the present invention relates to a picture data outputdevice that outputs data on an original picture for use in display. Thepicture data output device includes an original picture acquisitionsection that acquires data on the original picture, a tag datageneration section that generates tag data indicating the position of aparticular subject in the original picture, and a data output sectionthat outputs data on the original picture and the tag data inassociation with each other.

Still another aspect of the present invention relates to a contentcreation device. The content creation device includes a data acquisitionsection that acquires data on a plurality of original picturesphotographed from different viewpoints, and data on tags indicatingpositions of a particular subject in the respective original pictures, acontent generation section that generates video data using the originalpictures by setting data on an original picture for use in display, atiming of switching the original picture, and a visual field to bedisplayed, with respect to a time axis of the display, and a data outputsection that outputs the video data, in which the content generationsection controls a post-switching visual field at the timing ofswitching the original picture such that the position indicated by thetag for a same subject is included within the visual field before andafter switching.

Yet another aspect of the present invention relates to a contentreproduction method by a content reproduction device. The contentreproduction method includes a step of acquiring data on an originalpicture for use in display, and tag data indicating a position of aparticular subject in the original picture, a step of acquiring adisplay-time viewpoint and a display-time visual line with respect tothe original picture, a step of generating, from the original picture, adisplay picture in a visual field based on the display-time viewpointand the display-time visual line, and a step of outputting data on thedisplay picture to a display device, in which the step of generating thedisplay picture involves changing the display, in a state where apredetermined condition is satisfied, on the basis of the position inthe original picture indicated by the tag, by referring to the tag data.

A further aspect of the present invention relates to a picture dataoutput method by a picture data output device that outputs data on anoriginal picture for use in display. The picture data output methodincludes a step of acquiring data on the original picture, a step ofgenerating tag data indicating a position of a particular subject in theoriginal picture, and a step of outputting data on the original pictureand the tag data in association with each other.

A still further aspect of the present invention relates to a contentcreation method by a content creation device. The content creationmethod includes a step of acquiring data on a plurality of originalpictures photographed from different viewpoints, and data on tagsindicating positions of a particular subject in the respective originalpictures, a step of generating video data using the original pictures bysetting data on an original picture for use in display, a timing ofswitching the original picture, and a visual field to be displayed, withrespect to a time axis of the display, and a step of outputting thevideo data, in which the step of generating the video data involvescontrolling a post-switching visual field at the timing of switching theoriginal picture such that the position indicated by the tag for a samesubject is included within the visual field before and after theswitching.

It is to be noted that an optional combination of the aforementionedconstituent features, and any translation of the present invention to amethod, a device, a system, a computer program, a recording mediumhaving a computer program recorded therein, or the like, are alsoeffective as aspects of the present invention.

Advantageous Effect of Invention

According to the present invention, picture display in a preferablevisual field can be continued by using a whole-sky picture.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram depicting a configuration example of a contentprocessing system to which the present embodiment is applicable.

FIG. 2 is a diagram schematically depicting an example of a photographenvironment for acquiring an original picture according to the presentembodiment.

FIG. 3 is a diagram depicting an inner circuit configuration of apicture data output device according to the present embodiment.

FIG. 4 is a diagram depicting a functional block configuration of apicture data output device, a content generation device, and a contentreproduction device according to the present embodiment.

FIG. 5 is a diagram illustrating a relation between an original pictureand a space that includes photograph viewpoints and subjects accordingto the present embodiment.

FIG. 6 is a diagram schematically depicting an example of a method ofgenerating a display picture by using data on a plurality of originalpictures, data on the corresponding tags, and positional informationregarding the corresponding photograph viewpoints, according to thepresent embodiment.

FIG. 7 is an explanatory diagram of an aspect of suggesting thedirection in which a subject displayed before switching exists when dataon an original picture for use in display is switched, according to thepresent embodiment.

FIG. 8 is an explanatory diagram of an aspect in which the contentreproduction device changes a display-time visual line with respect toone original picture by using tag data according to the presentembodiment.

FIG. 9 is an explanatory diagram of a method of using tag data forcontent in which an actual situation in a place on a map is representedby photographed images, according to the present embodiment.

FIG. 10 is an explanatory diagram of a method of using tag data forcontent in which an actual situation in a place on a map is representedby photographed images, in the present embodiment.

FIG. 11 is a flowchart depicting process procedures in which the contentreproduction device according to the present embodiment generates adisplay picture by using a plurality of whole-sky pictures while usingtag data.

DESCRIPTION OF EMBODIMENT

FIG. 1 depicts a configuration example of a content processing system towhich the present embodiment is applicable. A content processing system1 includes a picture capturing device 12 that photographs a real space,a picture data output device 10 that outputs data on pictures, includinga photographed picture, for use in display, a content creation device 18that generates data on content involving picture display using anoutputted picture as an original picture, and a content reproductiondevice 20 that reproduces content involving picture display using anoriginal picture or content data.

A display device 16 a and an input device 14 a that are used by acontent creator to create content may be connected to the contentcreation device 18. A display device 16 b through which a content viewerviews a picture, and an input device 14 b through which an operation forthe content or for a displayed item is performed may be connected to thecontent reproduction device 20.

The picture data output device 10, the content creation device 18, andthe content reproduction device 20 establish communication thereamongover a wide-area communication network such as the internet or a localnetwork such as a LAN (Local Area Network). Alternatively, dataprovision from the picture data output device 10 to the content creationdevice 18 and the content reproduction device 20 and/or data provisionfrom the content creation device 18 to the content reproduction device20 may be carried out via a recording medium.

The picture data output device 10 and the picture capturing device 12may be connected via a wired cable or may be wirelessly connected over awireless LAN or the like. The content creation device 18 may beconnected to the display device 16 a and the input device 14 a by wireor wirelessly, and also, the content reproduction device 20 may beconnected to the display device 16 b and the input device 14 b by wireor wirelessly. Alternatively, two or more of the above devices may beintegrally formed. For example, the picture capturing device 12 and thepicture data output device 10 may be integrated into a picture capturingdevice or an electronic device.

The display device 16 b, on which a picture reproduced by the contentreproduction device 20 is displayed, does not need to be a flat display,and may be a projector or a wearable display such as a head mounteddisplay. The content reproduction device 20, the display device 16 b,and the input device 14 b may be integrated into a display device or aninformation processing device. Thus, no restriction is imposed on theouter shapes and connection forms of the depicted devices. In addition,in a case where the content reproduction device 20 directly processes anoriginal picture from the picture data output device 10 and generates adisplay picture, the content creation device 18 does not need to beincluded in the system.

The picture capturing device 12 includes a plurality of lenses 13 a, 13b, 13 c, 13 c, 13 d, 13 e, . . . , and a plurality of camerasrespectively including picture capturing sensors such as CMOS(Complementary Metal Oxide Semiconductor) sensors corresponding to thelenses. Each of the cameras takes a picture at an allocated angle ofview. The picture capturing device 12 has a mechanism of outputting atwo-dimensional luminance distribution of an image formed by lightcollected by the lenses, as in common cameras. Still pictured may bephotographed, or videos may be photographed.

The picture data output device 10 acquires data on photographed picturesoutputted from the cameras, and generates one original picture data setby connecting the photographed pictures. The term “original picture”refers to a picture that is an original a portion of which may bedisplayed or which is displayed after being processed. For example, in acase where a whole-sky picture is prepared, and a portion of thewhole-sky picture in a visual field corresponding to the visual line ofa viewer is displayed on a screen of a head mounted display, thiswhole-sky picture is regarded as an original picture. A whole-skypicture will be mainly used in the following explanation. However,original pictures in the present embodiment are not limited to whole-skypictures.

To obtain a whole-sky picture, the picture capturing device 12 thatincludes four cameras having respective optical axes at an interval of90 degrees with respect to a horizontal azimuth, and two cameras havingrespective optical axes extending vertically upward and downward isinstalled to photograph pictures each having an equal angle of viewwhich is a sixth part of the full azimuth. Then, in a picture plane suchas picture data 22 depicted in FIG. 1 in which the horizontal directionindicates an azimuth of 360° and the vertical direction indicates anazimuth of 180 degrees, photographed pictures are arranged in regionscorresponding to the angles of view of the respective cameras, and areconnected together. As a result, an original picture is generated. InFIG. 1, pictures photographed by the six cameras are denoted by “cam1”to “cam6,” respectively.

The format of the picture data 22 depicted in FIG. 1 is called anequirectangular projection and is generally used for drawing a whole-skypicture on a two-dimensional plane. However, the number of cameras andthe data format are not limited to those described above. The angle ofview of a picture obtained by the connection is also not limited to aparticular angle. In addition, it is general that a joint sectionbetween pictures is actually decided in view of the shape of an image inthe vicinity of the joint section. Thus, the joint section is notlimited to a straight line such as that depicted in FIG. 1. The picturedata 22 is compressed and encoded into a common format, and then, isprovided to the content creation device 18 via a network or a recordingmedium.

It is to be noted that, in a case where the picture capturing device 12photographs a video, the picture data output device 10 sequentiallygenerates and outputs the picture data 22 as an image frame at each timestep. The content creation device 18 creates content using the picturedata 22. This creation of content may be completely executed by thecontent creation device 18 on the basis of a prepared program or thelike, or at least a portion of the creation may be manually executed bya content creator.

For example, the content creator displays at least a portion of apicture indicated by the picture data 22 on the display device 16 a anddecides a region to be used for the content or associates the regionwith a reproduction program or an electronic game, through the inputdevice 14 a. Alternatively, the content creator may edit a videoindicated by the picture data 22 through a video edition application.Similar processing may be executed by the content creation device 18itself according to a previously created program or the like.

That is, as long as the picture data 22 is used, no limitation isimposed on the details or purpose of content to be created by thecontent creation device 18. Data on the content thus created is providedto the content reproduction device 20 via a network or a recordingmedium. Picture data included in the content may have the sameconfiguration as the picture data 22 or may have a data format and theangle of view that are different from those of the picture data 22.Pictures having undergone a certain processing may also be used.

The content reproduction device 20 displays a picture for the content onthe display device 16 b by, for example, performing informationprocessing provided as the content, in response to a content viewer'soperation on the input device 14 b. According to the type of thecontent, a viewpoint or a visual line with respect to a display picturemay be changed in response to a viewer's operation on the input device14 b. Alternatively, such a viewpoint or a visual line may be determinedby the content side.

By way of example, the content reproduction device 20 maps, on the innersurface of a celestial sphere centered on a content viewer with a headmounted display, the picture data 22 that is obtained by photographingfrom the position of the content viewer, and displays, on a screen ofthe head mounted display, a picture for a region toward which the faceof the content viewer is directed. Accordingly, no matter whichdirection the content viewer faces, the content viewer can see a pictureworld in a visual field corresponding to the direction. Thus, thecontent viewer can feel as if the viewer entered the picture world. Inthis case, a motion sensor included in the head mounted display servesas the input device 14 b, and acquires, as input information, ameasurement value for deriving which direction the content viewer faces.

Alternatively, a flat display may be used as the display device 16 b,and the content viewer moves a cursor displayed on the flat display suchthat scenery in the direction to which the cursor is moved can be seen.It is to be noted that, when it is not necessary to edit a picture orassociate a picture with another information, the content reproductiondevice 20 may directly acquire the picture data 22 from the picture dataoutput device 10 and may display the entirety or a portion of thepicture data 22 on the display device 16 b.

FIG. 2 schematically depicts an example of a photograph environment foracquiring an original picture according to the present embodiment. Inthe present embodiment, whole-sky pictures are photographed from aplurality of viewpoints so that a plurality of original pictures fromdifferent viewpoints are acquired. For example, a space including asubject 80 is photographed from the viewpoints of picture capturingdevices 12 a, 12 b, and 12 c, as illustrated in FIG. 2. It is to benoted that a plurality of the picture capturing devices 12 a, 12 b, and12 c are not necessarily installed to perform photographingsimultaneously. Alternatively, one picture capturing device may be movedto the viewpoints and perform photographing at independent timings. Inaddition, any limitation is not imposed on the number of the viewpoints.Hereinafter, a viewpoint from which an original picture is photographedis referred to as “photograph viewpoint.”

In the aforementioned manner, original pictures photographed fromdifferent photograph viewpoints are prepared to allow a content creatorto switch the composition of a video, or to switch an original picturefor use in display according to a viewpoint or visual line desired by aviewer. For example, it is common to, in the process of reproducing avideo such as a movie, perform switching to a picture of the same spaceat a different angle such that the expression becomes effective or thefacial expression of a player becomes clearer.

However, in a case where content such as a video is created by usingpreviously photographed whole-sky pictures, as in the presentembodiment, when a photograph viewpoint or an original picture for useis switched, setting of the visual field before and after the switchingis difficult. In the example in FIG. 2, the azimuth in which the subject80 is present varies with respect to the viewpoints of the picturecapturing devices 12 a, 12 b, and 12 c. Thus, in picture data 82 a, 82b, and 82 c which are equirectangular projections obtained from therespective visual points, the position of an image of the subject 80naturally varies.

To perform display switch from a state in which a picture having avisual field of a region 84 a is represented by use of the picture data82 a, to a state in which a picture using the picture data 82 b isdisplayed, for example, the most simple way may be displaying a region84 b the range of which is unchanged irrespective of the visual point(the azimuth of which is the same with respect to the visual point).However, in a case where a viewer is watching an image of the subject 80before switching, the image of the subject 80 is not included in theregion 84 bb which is displayed after the switching. Accordingly,confusion can be caused so that the visual line is not settled.

A content creator who creates the video may set original pictureswitching, and also set the visual field to display pictures withcontinuity. However, much labor is required to find a proper visualfield from a wide-angle picture such as a whole-sky picture. Inaddition, the intervention of the content creator is indispensable. Notonly in a case of previously performing switching for a video, but alsoin a case of switching original pictures according to a viewer'soperation, a viewer may lose a target which the viewer desires to seeand need to find out the target from a wide angle of view. This gives abig stress to the viewer.

To this end, the picture data output device 10 according to the presentembodiment provides not only original pictures from a plurality ofphotograph viewpoints, but also data on tags indicating positions of aparticular subject in the planes of the respective original pictures.For example, in the planes of the picture data 82 a, 82 b, and 82 c,tags are respectively provided to regions 86 a, 86 b, and 86 c. Then,the visual line is controlled such that, in a case where any one of thetagged regions is included in the displayed visual field immediatelybefore original picture switching, the tagged region is still includedin the visual field of the switched original picture.

In the example depicted in FIG. 2, since the tagged region 86 a isincluded in the region 84 a of the picture data 82 a which is displayedbefore switching, a region 84 c including the tagged region 86 b in theswitched picture data 82 b is displayed. In a case where there is aplurality of particular subjects, identification information foridentifying each subject is further included in a tag. Thus, the samesubject is displayed before and after switching. It is to be noted thata particular subject refers to a subject that is detected in accordancewith a predetermined rule, or a subject that is selected by a person. Inthis manner, a displayed subject can be inherited, irrespective ofswitching of original pictures for use in display. As a result, whilethe range of picture expression is widened or a providable informationamount is increased by means of a multi-viewpoint whole-sky picture,dissipated presentation can be easily avoided. Thus, the visibility andconvenience can be enhanced.

FIG. 3 depicts an inner circuit configuration of the picture data outputdevice 10. The picture data output device 10 includes a CPU (CentralProcessing Unit) 23, a GPU (Graphics Processing Unit) 124, and a mainmemory 26. These components are mutually connected via a bus 30. Aninput/output interface 28 is also connected to the bus 30. Acommunication section 32 that is including a peripheral device interfacesuch as a USB (Universal Serial Bus) or IEEE1394 or of a networkinterface for a wired or wireless LAN, a storage section 34 that is ahard disk drive, a nonvolatile memory, or the like, an output section 36that outputs data to an external device, an input section 38 thatreceives data from an external device such as the picture capturingdevice 12, and a recording-medium driving section 40 that drives aremovable recording medium such as a magnetic disc, an optical disk, ora semiconductor memory, are connected to the input/output interface 28.

The CPU 23 controls the entirety of the picture data output device 10 byexecuting the operating system stored in the storage section 34. The CPU23 further executes various programs that are read out from a removablerecording medium and are loaded into the main memory 26, or that aredownloaded via the communication section 32. The GPU 24 has a functionas a geometry engine and a function as a rendering processor, andperforms a rendering in accordance with a rendering command from the CPU23, and gives an output to the output section 36. The main memory 26includes a RAM (Random Access Memory) and stores a program or datanecessary for processing. It is to be noted that each of the contentcreation device 18 and the content reproduction device 20 may also havethe similar inner circuit configuration.

FIG. 4 depicts a functional block configuration of the picture dataoutput device 10, the content creation device 18, and the contentreproduction device 20. The functional blocks depicted in FIG. 4 can beimplemented by the circuits depicted in FIG. 3 in terms of hardware, andcan be implemented by a program loaded from a recording medium to a mainmemory so as to exhibit various functions including an image analysisfunction, an information processing function, an image renderingfunction, and a data input/output function, in terms of software.Therefore, it should be understood by a person skilled in the art thatthese functional blocks can be implemented in various forms, that is, byhardware only, software only, or a combination thereof. No limitation isimposed on this combination.

The picture data output device 10 includes an original pictureacquisition section 50 that acquires data on a plurality of originalpictures, a tag data generation section 52 that generates data on tagscorresponding to the original pictures, and a data output section 54that outputs data on the original pictures and the corresponding tags,and positional information regarding the photograph viewpoints. Theoriginal picture acquisition section 50 is implemented by the inputsection 38, the CPU 23, the GPU 24, the main memory 26 in FIG. 3, andthe like, and acquires data on pictures photographed from a plurality ofviewpoints by the picture capturing device 12, and information regardingthe positions where the pictures have been photographed.

In a case where the picture capturing device 12 photographs pictures bydividing the angle of view, as depicted in FIG. 1, the original pictureacquisition section 50 acquires the photographed pictures, connects thepictures together, and generates, as an original picture, one whole-skypicture. In a case where the pictures already have been connectedtogether at the picture capturing device 12, the original pictureacquisition section 50 may directly use, as an original picture, thephotographed picture acquired from the picture capturing device 12. Itis to be noted that the original picture acquisition section 50 mayacquire data on a whole-sky picture that is rendered by computergraphics, instead of a photographed picture.

A photographed picture is used in the following explanation, but thisexplanation can also be applied to a picture obtained by computergraphics. The original picture acquisition section 50 supplies, to thedata output section 54, data on the acquired original pictures and dataindicating a positional relation among the viewpoints where the pictureshave been photographed. Further, the data on the original pictures isalso supplied to the tag data generation section 52.

The tag data generation section 52 is implemented by the CPU 23, the GPU24, the main memory 26 in FIG. 3, and the like, and detects a subjectimage from the original picture by a predetermined algorithm, andgenerates tag data indicating the position of the subject image in thepicture plane. Various technologies for detecting a specific subjectimage from a picture have been put into actual use. The presentembodiment can adopt any one of these technologies. Examples of thetechnologies include matching using a template subject image, and facedetection using a feature amount of an average face. In a case where thesubject is moving, a feature-point following technology using an opticalflow or the like can be adopted. Further, through deep learning, asubject may be surmised from the shape or color of the subject image.

The tag data generation section 52 derives, as an “image position” to betagged, information regarding the position coordinates of arepresentative point of a detected image or about a predetermined rangeregion including the image in the plane of each original picture (e.g.the plane of an equirectangular projection). The representative pointrefers to a point that is included in an image and that is determined inaccordance with a predetermined rule, and the point is the gravitycenter of the image, an upper left point, or the like. The predeterminedrange region including the image refers to the region of the imageitself, the circumscribed rectangle of the image, a predetermined sizerectangle including the image, or the like. Information regarding theregion may be map data in which variation of pixel values is indicatedin a picture plane, or may be data indicating the position and size ofthe region by numerical values, such as the coordinates of the upperleft position of the circumscribed rectangle and two adjacent sides ofthe circumscribed rectangle.

In a case where the above image detection processing indicates that aplurality of particular subjects are included, the tag data generationsection 52 generates tag data by associating the positions of imageswith identification information regarding the corresponding subjects. Asubject to be tagged may be selectable from among various subjectsincluded in the picture. For example, the tag data generation section 52may display, on a display device (not depicted), an original picture inwhich the images of detected subjects are enclosed with geometricalfigures to allow a user to select a target to be tagged.

Alternatively, the original picture may be displayed as it is such thata user is allowed to initially designate a subject to be tagged. In thiscase, the tag data generation section 52 can omit a process of detectingsubject images. In a case where the original picture is a video,information in the tag changes with movement of a subject, as a matterof course. Therefore, the tag data generation section 52 generates tagdata for each frame of the original picture.

The data output section 54 is implemented by the CPU 23, the main memory26, the communication section 32 in FIG. 3, and the like, and outputs aplurality of original pictures, data on the corresponding tags, and dataon positional information regarding the viewpoints to the contentcreation device 18 or the content reproduction device 20. Alternatively,the data output section 54 may include the recording-medium drivingsection 40 to store the above data in a recording medium. It is to benoted that, in a case where the picture data is a video, the data outputsection 54 outputs each frame constituting the video and tag data inassociation with each other. In addition, the picture data is compressedand encoded, as appropriate, before being outputted.

The content creation device 18 includes a data acquisition section 60that acquires data on a plurality of original pictures and thecorresponding tags, and data on positional information regardingviewpoints, a data storage section 62 that stores the above data, acontent generation section 64 that generates content using data on theoriginal pictures, and a data output section 66 that outputs data on thecontent. The data acquisition section 60 is implemented by thecommunication section 32, the CPU 23, the main memory 26 in FIG. 3, andthe like, and acquires various data outputted from the picture dataoutput device 10. Alternatively, the data acquisition section 60 mayinclude the recording-medium driving section 40 to read out the abovedata from a recording medium, as previously explained.

The data acquisition section 60 decodes and decompresses the acquireddata, as appropriate, and stores the decoded and decompressed data inthe data storage section 62. The data storage section 62 is implementedby the main memory 26 in FIG. 3, and the like, and stores data on aplurality of original pictures taken from different photographviewpoints and tags corresponding to the original pictures, and data onpositional information regarding the photograph viewpoints. The contentgeneration section 64 is implemented by the CPU 23, the GPU 24, the mainmemory 26 in FIG. 3, and the like, and generates data on contentinvolving picture display by using the data stored in the data storagesection 62.

Content to be created is an electronic game, a video for appreciation,or an electronic map, for example. No limitation is imposed on the genreand purpose of the content. A picture to be included in the content maybe the whole original picture data or may be a portion thereof.Information for defining a way to select and display the picture may beautomatically created by the content generation section 64, or may be atleast partially, manually created by a content creator. In the lattercase, the content generation section 64 further includes an outputsection 36 and an input section 38, and displays a picture on thedisplay device 16 a, and then, receives a request to edit the picture orgenerate various information, from the content creator via the inputdevice 14 a.

In any case, the content generation section 64 generates data on thecontent for implementing either one of the following two display modes.

(1) A video in which original pictures and visual fields thereof thatare previously defined are used.

(2) A picture in which original pictures and visual fields thereof thatare changed according to a content viewer's operation are used.

In the case of (1), the content generation section 64 sets data on anoriginal picture for use in display and a timing of switching theoriginal picture with respect to a time axis of the display.Furthermore, the content generation section 64 sets a temporal change ofa visual field to be displayed in the original picture. For example, inthe subject field depicted in FIG. 2, the content generation section 64determines the subject 80 to be displayed or an angle thereof. Aphotograph viewpoint that enables the desired display is selected on thebasis of positional information regarding photograph viewpoints storedin the data storage section 62. Accordingly, setting is performed suchthat an original picture corresponding to the selected photographviewpoint is used in display.

Furthermore, among the original pictures, an original picture having aproper visual field is determined. In a case where an original picturethat is optimum for use in display is shifted due to movement of asubject, the content generation section 64 determines a timing ofswitching the original picture. Here, the content generation section 64controls a post-switching visual line or visual field such that aposition provided with a tag for the same subject is included in thevisual field before and after the switching. Accordingly, video contentin which the continuity of a displayed subject is maintained even afteran original picture is switched, can be easily generated. In addition,the display angle of the same subject is changed through the originalpicture switching so that an effect of expression is exhibited.

In the case of (2), the content generation section 64 generates, on thebasis of what is instructed by a content viewer's operation, dataindicating a rule for deriving an original picture to be displayed and avisual field thereof. Here, the content viewer's operation is notlimited to operations for directly operating viewpoints and visuallines, and may be an indirect operation, such as any command for anelectronic game, that leads to a viewpoint change or a visual linechange. That is, this rule may be a program for an electronic game. Inthe above cases, the content generation section 64 generates, as thecontent data, data indicating the above rule, data on a plurality oforiginal pictures and tags corresponding to the original pictures, andpositional information regarding the photograph viewpoints.

Depending on the display purpose, the content generation section 64 maygenerate new tag data for each original picture. For example, only tagdata that is necessary for the purpose, among the tags generated by thepicture data output device 10, may be left, or new tag data may begenerated again from an original picture. In this case, the contentgeneration section 64 may automatically provide a tag in accordance witha predetermined rule, or a content creator may manually at least aportion of this processing, as in the tag data generation section 52 ofthe picture data output device 10.

Further, the content generation section 64 may process an originalpicture or select a necessary original picture. The data output section66 is implemented by the CPU 23, the main memory 26, the communicationsection 32 in FIG. 3, and the like, and compresses and encodes thecontent data generated by the content generation section 64, asappropriate, and outputs the data to the content reproduction device 20.Alternatively, the data output section 66 may include therecording-medium driving section 40 to store the content data in arecording medium.

The content reproduction device 20 includes a data acquisition section70 that acquires various data from the picture data output device 10 orthe content creation device 18, a data storage section 72 that storesthe acquired data, a display-viewpoint acquisition section 74 thatacquires a display-time viewpoint and a display-time visual lineaccording to a content viewer's operation, a display-picture generationsection 76 that generates a picture corresponding to the viewpoint andthe visual line, and a data output section 78 that outputs data on adisplay picture. The data acquisition section 70 is implemented by thecommunication section 32, the CPU 23, the main memory 26, etc. in FIG.3, and acquires data on a plurality of original pictures and tagscorresponding to the original pictures, and data on positionalinformation regarding the photograph viewpoints, outputted from thepicture data output device 10, or acquires content data outputted fromthe content creation device 18. Alternatively, the data acquisitionsection 70 may include the recording-medium driving section 40 to readthe above data from a recording medium.

The data acquisition section 70 decodes and decompresses the acquireddata, as appropriate, and stores the data in the data storage section72. The data storage section 72 is implemented by the main memory 26,etc. in FIG. 3, and stores data on the plurality of original picturesand the corresponding tags, and data on positional information regardingthe photograph viewpoints, or stores the content data. Thedisplay-viewpoint acquisition section 74 is implemented by the CPU 23,the GPU 24, the input section 38, etc. in FIG. 3, and acquires a visualpoint or visual line corresponding to a content viewer's operation viathe input device 14 b.

For example, display of a GUI (Graphical User Interface) such as acursor for operating the position of a viewpoint or the direction of avisual line is superimposed on a picture being displayed such that thevisual point and the visual line are changed according to an operationperformed on the GUI. Alternatively, a game is advanced in response toan input of a command for an electronic game, and a viewpoint and avisual line are decided according to the command. It is to be notedthat, in a case where a video created by the content creation device 18is displayed, the function of the display-viewpoint acquisition section74 can be omitted if a content viewer does not need to operate theviewpoint and the visual line.

The display-picture generation section 76 is implemented by the CPU 23,the GPU 24, the main memory 26 in FIG. 3, and the like, and generates apicture to be displayed on the display device 16 b by using data storedin the data storage section 72. In a case where a video created by thecontent creation device 18 is displayed, picture frames obtained bydecoding and decompression may be directly outputted. In a case where acontent viewer is allowed to operate the viewpoint and the visual line,the display-picture generation section 76 decides an original picturefor use in display and a region to display the original picture,according to the viewpoint and visual line acquired by thedisplay-viewpoint acquisition section 74.

That is, the display-picture generation section 76 selects an originalpicture photographed from a photograph viewpoint that corresponds to thedisplay-time viewpoint, by referring to the positional informationregarding photograph viewpoints stored in the data storage section 72,and further, decides a region to display the original picture. In a casewhere the display-time viewpoint or the subject is movable, thedisplay-picture generation section 76 selects the correspondingphotograph viewpoint and switches an original picture for use. Then, ina state where a predetermined condition is satisfied, the display ischanged on the basis of the position provided with the tag, by referringto the tag data. Specifically, as previously explained, when originalpicture switching is performed, a visual line with respect to a switchedoriginal picture is controlled such that a position provided with a tagfor the same subject is included in the visual field before and afterthe switching.

Alternatively, as explained later, when predetermined input means isoperated by a content viewer, the visual line is controlled such that aposition provided with a tag is included in the visual field. Then, ateach time step, a picture corresponding to the decided visual field isgenerated as a display picture. It is to be noted that, in a state wherea predetermined condition is satisfied as explained above, thedisplay-picture generation section 76 may display a geometrical figureindicating the relative direction of a tagged position, instead ofdirecting the visual line toward the tagged position. The data outputsection 78 is implemented by the CPU 23, the main memory 26, the outputsection 36, etc. in FIG. 2, and outputs data on the display picturegenerated in the aforementioned manner to the display device 16 b. Thedata output section 78 may output sound data, if needed, in addition todata on the display picture.

FIG. 5 illustrates a relation between an original picture and a spaceincluding photograph viewpoints and subjects. The left side in FIG. 5depicts a state where a space including subjects 80 and 81 is seen froma bird's-eye view. Data indicating a positional relation betweenphotograph viewpoints 102 a and 102 b in this space isphotograph-viewpoint positional information 100. It is to be noted thatthe photograph-viewpoint positional information 100 actually indicatesthe position coordinates of the photograph viewpoints 102 a and 102 b ina two-dimensional or three-dimensional space. If videos are respectivelyphotographed from the photograph viewpoints 102 a and 102 b, the picturedata output device 10 associates with data on time-series originalpictures 104 and 105 representing picture frames of the videos, with thepositional information 100 regarding the photograph viewpoints 102 a and102 b, and outputs the data, as depicted on the right side in FIG. 5.

An azimuth is previously defined for each of the photograph viewpoints102 a and 102 b. In the example depicted in FIG. 5, the upper side isdefined as 0°, and the azimuth (longitude) in the horizontal directionis defined so as to have an angle that increases in the clockwisedirection. The azimuth (latitude) in the vertical direction is definedso as to have an angle the range of which is −90° to 90° with respect tothe horizon which is defined as 0°. In the original pictures 104 and105, images (e.g., images 180, 181) of the subjects 80 and 81respectively observed from the respective photograph viewpoints 102 aand 102 b are displayed at positions corresponding to the azimuths. Inthe example depicted in FIG. 5, the center in the lateral direction ineach of the original pictures 104 and 105 is defined as a longitude of0°, and an image in an azimuth from the center to the right direction by180°, and an image in an azimuth of 180° to 360° (=0°), which extendsfrom the left end of the picture to the center are displayed, whereby afull surround picture is obtained.

However, the definitions of the azimuths and the expression in anoriginal picture are not limited to those in FIG. 5. When a picture isdisplayed at the content reproduction device 20, the direction of thevisual line is decided with the photograph viewpoint 102 a set, asappropriate, for example, so that a region 112 corresponding to thedirection in the original picture 104 is displayed. In this example, apicture mainly including the image 181 of the subject 81 is displayed.This display mode, in which the direction of a visual line ischangeable, is regarded as 3DOF (3 Degrees of Freedom). However,elements (subjects) constituting a photographed scene arestereophonically reproduced by use of a technology such as SfM(Structure from Motion) or camera mapping, a display-time viewpoint canbe freely moved.

That is, 6DOF (6 Degrees of Freedom) reproduction in which a displaypicture is generated with a viewpoint set at a position different from aphotograph viewpoint, can be performed. Accordingly, a process oforiginal picture switching to a different photograph viewpoint, whichwill be explained later, can be performed, even according to movement ofa display viewpoint which is controlled independently of a photographviewpoint. In this case, a rule to select a photograph viewpoint that isclosest to a display viewpoint, for example, is introduced.

FIG. 6 schematically depicts an example of a method for generating adisplay picture by using data on a plurality of original pictures andtags corresponding to the original pictures, and positional informationregarding the photograph viewpoints. In this example, it is assumed thatthe subject 80 is standing still while the subject 81 is moving along atrack indicated by a dotted arrow, as depicted in a bird's eye view onthe left side. The center in FIG. 6 depicts that some frames of videosphotographed from the photograph viewpoints 102 a and 102 b aredisplayed as original pictures 104 a, 104 b, 105, and 104 c.Specifically, the original pictures 104 a, 104 b, and 104 c arephotographed from the photograph viewpoint 102 a at times t1, t2, and t4in a time axis, which is depicted on the right side of FIG. 6, and theoriginal picture 105 is photographed from the photograph viewpoint 102 bat time t3.

Since the subject 80 is standing still, images 180 a, 180 b, and 180 dof the subject 80 in the original pictures 104 a, 104 b, and 104 c,which are photographed from the same photograph viewpoint 102 a, aredisplayed at the same position. Since the subject 81 is moving, images181 a, 181 b, etc. of the subject 81 are moving in the planes of theoriginal pictures 104 a, 104 b, etc., which are photographed from thesame photograph viewpoint 102 a through. In the original picture 105photographed from the different photograph viewpoint 102 b, images 180 cand 181 c of both the subjects 80 and 81 are displayed at differentpositions.

The tag data generation section 52 of the picture data output device 10generates tag data for each video frame photographed from the photographviewpoints. For example, tag data 106 a, 106 b, 107, and 106 c aregenerated for the original pictures 104 a, 104 b, 105, and 104 c,respectively. The tag data 106 a, 106 b, 107, and 106 c in FIG. 6 aremap data obtained by giving different pixel values to circumscribedrectangles of the regions of the images of the subjects 80 and 81 in therespective image planes of the corresponding original pictures 104 a,104 b, 105, and 104 c.

For example, in the original pictures 104 a, 104 b, and 105,circumscribed rectangles 182 a, 182 b, 182 c, 184 a, 184 b, and 184 c ofthe images 180 a, 180 b, 180 c, 181 a, 181 b, and 181 c are coloredblack, and the remaining regions are colored white. However, the formatof tag data is not limited to this example, as explained above, and theposition coordinates of the gravity centers of the regions of the imagesof the subjects 80 and 81 may be adopted. In addition, in a case wherethere are a plurality of subjects to be tagged, as depicted in FIG. 6,identification information for identifying each subject is associatedwith the corresponding tag.

In a case where the subject 81 is a main display target, the subject 81is located close to the photograph viewpoint 102 a at time t1.Therefore, the original picture 104 a photographed from the photographviewpoint 102 a is selected, and a picture of the region 112 a includingthe image 181 a of the subject 81 in the original picture 104 a isdisplayed. Since the subject 81 is still located close to the photographviewpoint 102 a at time t2, the original picture 104 b photographed fromthe photograph viewpoint 102 a is selected, and a picture of the region112 a including the image 181 b of the subject 81 in the originalpicture 104 b is displayed. In these stages, the visual line is decidedin accordance with a predetermined rule in the content creation device18 or the content reproduction device 20, or is decided by a contentcreator or a content viewer.

In a case where the viewpoint is changed to the photograph viewpoint 102b, which is closest at time t3, due to movement of the subject 81, theoriginal picture for use in display is switched to the original picture105 photographed from the photograph viewpoint 102 b. Here, the contentgeneration section 64 of the content creation device 18 or thedisplay-picture generation section 76 of the content reproduction device20 confirms whether or not a tagged position is included in the region112 b which has been displayed before the switching, by referring to thetag data 106 b corresponding to the original picture 104 b before theswitching.

In a case where a tagged position is included, a position provided witha tag for the same subject is acquired by referring to the tag data 107that corresponds to the original picture 105 after the switching. In theexample depicted in FIG. 6, the position of the tag 184 c for the samesubject as the tag 184 b which is included in the region 112 b displayedbefore the switching is acquired. Then, the region 112 c including thisposition is decided as a target to be displayed after the originalpicture switching. This process is performed to adjust the visual linefrom a post-switching viewpoint such that the subject 81 is included inthe visual field. In response to the subsequent movement of the subject81, the visual field may be set according to the set visual line. It isto be noted that, basically in this example, an original picture isswitched in the aforementioned manner at a time point when the closestphotograph viewpoint is shifted according to movement of the subject 81,whereby the visual field is properly controlled.

On the other hand, particularly in a case where video content iscreated, a viewpoint may be jumped to a remote position (e.g. photographviewpoint 102 c) such that display from a greatly changed angle isimplemented to produce an effect of expression. Also, in this case, atagged position is included in the visual field when an original pictureis switched so that the continuity of the display is maintained.Further, this saves a content creator from having to find out a propervisual field.

Further, in the example depicted in FIG. 6, when the region 112 c in theoriginal picture 105 is displayed, the image 180 c of the subject 80appears at the left end of the picture. In this case, through theappearance of the image, an attention is attracted to improve thesaliency. Accordingly, an effect of motivating a viewer to watch thesubject 80 from the opposite direction is expected. Therefore, thephotograph viewpoint for an original picture for use in display can beintentionally switched by the viewer operating any input means of theinput device 14 b. For example, when the viewpoint is returned from thephotograph viewpoint 102 b to the photograph viewpoint 102 a, thepicture for use in display is switched from the original picture 105 tothe original picture 104 c, as depicted in the transition from time t3to t4.

In this case, the display-viewpoint acquisition section 74 alsoreceives, from the viewer, an operation of switching the target subjectfrom the subject 81 to the subject 80. Specifically, the image 180 c ofthe subject 80 is made selectable in a state where the region 112 c isbeing displayed. Accordingly, the display-picture generation section 76decides a region 112 d to be displayed so as to include the position ofa tag 182 d for the subject 80, by referring to the tag data 106 ccorresponding to the switched original picture 104 c.

As a result, an image 180 d of the subject 80, which has attracted a newattention, viewed from a different angle is displayed. It is to be notedthat there are various methods for deciding which one of the tags for aplurality of the subjects is considered to be effective. For example,buttons provided on the input device 14 b may be allocated to respectivesubjects, and a tag for a subject corresponding to a pressed button maybecome effective. Alternatively, a tag for a subject that is displayedat the screen center immediately before viewpoint switching may becomeeffective.

In a case where any tagged position is not included in the visual fieldbefore the viewpoint switching, the visual line may be decided to thesame azimuth, when viewed from the photograph viewpoint before and afterthe switching, for example, and a picture having the correspondingvisual field may be displayed. Alternatively, the visual line may bedecided toward the same azimuth in the space, and a picture having thecorresponding visual field may be displayed. In an embodiment in which acontent viewer operates the viewpoint and the visual line, when originalpicture data is switched, not a region, such as the region 112 c,including a tagged position is directly displayed, but auxiliary dataindicating a direction in which a subject image is present may bedisplayed so that the image is displayed according to a content viewer'soperation.

FIG. 7 is an explanatory diagram of an embodiment in which, when data onan original picture for use in display is switched, a direction in whicha subject displayed before the switching is present is indicated.Similarly to the original picture 105 in FIG. 6, an original picture 120represents an original picture which is used after switching. In thisembodiment, the display-picture generation section 76 of the contentreproduction device 20 does not directly display a region including atagged position in tag data 122, a picture of a region 124 which is inthe same azimuth as that before switching is displayed also after theswitching, for example. In this example, a subject image provided with atag is outside the visual field, and thus, is not displayed, as depictedin a display picture 126 on the lower side.

Here, the display-picture generation section 76 detects that a taggedposition included in the display region before switching is not includedin the display region 124 after switching, by referring to tag data 122that is associated with the original picture 120. In this case, thedisplay-picture generation section 76 superimposes, on the displaypicture 126, display of an arrow 114 that indicates the direction of thetagged subject with respect to the current display region 124.

Accordingly, if a content viewer wants to continuously view the subjectthat the content viewer has viewed before switching, the content viewercan easily obtain the desired visual field by changing the visual linetoward the direction of the arrow 114 via the input device 14 b. It isto be noted that the direction of the arrow 114 to be displayed does notnecessarily match the direction from the region 124 toward the subjectin the original picture 120, and thus, is decided, as appropriate,according to a positional relation between the photograph viewpoint andthe subject in the photograph space.

In the embodiment explained so far, when an original picture is switchedaccording to movement of a visual line, a subject that is displayedbefore the switching is displayed even after the switching. However, tagdata may be introduced in order to easily include an important subjectimage in a visual field in a whole-sky picture, irrespective of originalpicture switching. This method is also implemented by the contentreproduction device 20, in an embodiment in which a content vieweroperates a viewpoint and a visual line. FIG. 8 is an explanatory diagramof an embodiment in which the content reproduction device 20 changes adisplay-time visual line with respect to one original picture by usingtag data.

FIG. 8 depicts a state where a whole-sky original picture 130 is seenfrom a bird's eye view. As in the embodiments previously explained, anequirectangular projection can be adopted as the data format. However,in FIG. 8, an axis of the longitudinal direction is depicted in thecircumferential direction. It is assumed that, in the plane of theoriginal picture 130, a region 132 is provided with a tag. The tag isprovided in the content creation device 18 by a content creator. The tagis provided to a region, in the original picture, including a mainperson or object or a region that the content creator particularly wantsto display, for example.

On the other hand, in the content reproduction device 20, a picture isdisplayed with use of the original picture 130. For example, it isassumed that a content viewer displays a visual field 136 a thatcorresponds to a visual line 134 a according to the content viewer'soperation. In a case where the main object is not included in the visualfield 136 a, the content viewer desires to restore the display of theregion including the main object. However, in a 360° whole-sky picture,a movable range is too wide to easily restore display in some cases.

To this end, when the content viewer operates specific input means ofthe input device 14 b, display is immediately changed to a visual fieldincluding a tagged region. Specifically, when the input means of thedisplay-viewpoint acquisition section 74 is operated, thedisplay-viewpoint acquisition section 74 of the content reproductiondevice 20 changes the visual line 134 a to the visual line 134 b so asto obtain a visual field 136 b including the tagged region 132, byreferring to the tag data. Accordingly, a portion, of a wide picture,that the content creator wants to display, that is, a portion that thecontent viewer wants to view can be easily preferentially displayed.

One original picture is depicted in FIG. 8, but similar tag data can beassociated with all the original pictures for use in display, as in theaforementioned examples, and thus, whichever original picture is used,the main portion can be easily displayed. It is to be noted that FIG. 8depicts the state in which the display viewpoint matches the photographviewpoint which is at the center of the original picture 130. However,the same applies to a state in which the display viewpoint is displacedfrom the photograph viewpoint. That is, a visual line vector from thedisplay viewpoint to the tagged region 132 can be easily derived on thebasis of the amount of displacement from the photograph viewpoint andthe direction of the displacement. Also, in a case where an originalpicture is a video, tag data is changed according to a shift of the mainportion, as a matter of course. Accordingly, when not the visual fieldis changed, but a main subject goes outside the visual field, forexample, the display can follow the subject.

In addition, in a case where content is a mixture of a picture and asound, a tag is provided, simultaneously with occurrence of a sound, toa subject that is the occurrence source of the sound. Accordingly, thevisual line can be directed toward the direction of the sound,irrespective of the direction of the displayed visual field. As aresult, a desire to confirm the occurrence source of the sound can besatisfied without any stress. In either case, since a content viewerstarts to move the visual line on the condition that the content viewerhas a desire to move the visual line so that motion sickness, which islikely to be induced by an automatic visual field change in a headmounted display, is unlikely induced. The embodiment depicted in FIG. 8may be combined with the visual line control in original pictureswitching or may be independently introduced. In the former case, commontag data may be used, or separate tag data may be independentlygenerated.

FIGS. 9 and 10 are explanatory diagrams of methods of using tag data incontent for depicting an actual site at a point on a map by using aphotographed picture. First, FIG. 9 schematically depicts a generalcontent operation method and a display transition of the content. Adisplay screen 140 a displays a picture actually photographed at apreviously designated site on a map. In this example, a geometrical FIG.148a that represents a designated current position and a photographedpicture including a road and a building 146 that can be seen from thecurrent position are displayed.

This picture actually displays a partial region 154 a of an originalpicture 142 a, which is a whole-sky picture, as indicated by a dottedline. A visual line for deciding the region 154 a can be changed by useof the input device 14 b. The left side in FIG. 9 indicates a statewhere the positional relation between a photograph viewpoint 152 a ofthe original picture 142 a or the like and the building 146 is seen froma bird's eye view. It is assumed that the viewpoint of a picture beingdisplayed in the display screen 140 a matches the photograph viewpoint152 a. That is, the building 146 is included on the right front side ofa visual field that corresponds to the visual line in the direction ofan arrow A from the photograph viewpoint 152 a.

The display screen 140 a further displays an arrow 150 for designating amovement direction. To confirm the building 146 located on the frontside, the content viewer directs and determines the arrow 150 by using acursor or the like through the input device 14 b so that the viewpointcan be moved forward. However, as a result of this operation, theviewpoint is frequently moved too far, as depicted in a display screen140 b, so that the building 146 which is the target goes outside thevisual field.

That is, when the viewpoint is moved toward the direction of the arrowA, the display is switched to display using a next original picture 142b photographed from a photograph viewpoint 152 b. In a case where anytag data is not used, the direction of the visual line remains in thedirection of the arrow A which indicates the advance direction, andthus, the building 146 is not included in a corresponding visual fieldregion 154 b in the original picture 142 b. In this case, the contentviewer has to find out the building 146 by changing the direction of thevisual line. In addition, when the building 146 suddenly disappears fromthe display, the content viewer cannot intuitively get to understandwhich location is being displayed, and may feel stress.

Therefore, in map content involving display of actually photographedpictures, a landmark such as a building is often considered to be moreimportant than following a road. FIG. 10 schematically depicts a displaytransition when a landmark is provided with a tag to control a visualline. On a display screen 160 a, a photographed picture when thedirection of the visual line is determined as the direction of an arrowA from the photograph viewpoint 152 a is displayed, as on the displayscreen 140 a in FIG. 9. The building 146 is displayed on the right frontside.

On the other hand, in each of the original pictures 142 a and 142 bphotographed from the photograph viewpoints 152 a and 152 b, a tag isprovided to the region of an image of an object which is a landmark suchas the building 146, and tag data 162 a and 162 b indicating the tagposition is associated with the tag. The target of the tag is notlimited to any particular type as long as the target is an object, suchas a building, a scenic spot, or a large street intersection, that theviewer is highly likely to set as a target. As previously explained, inthe picture data output device 10 or the content creation device 18, thetag is provided on the basis of automatic detection of an image ordesignation made by a content creator.

In a case where a tagged position is included in the region 154 a beingdisplayed, the display-viewpoint acquisition section 74 of the contentreproduction device 20 allows the content viewer to select the position.For example, display of an arrow 164 indicating the building 146provided with a tag is superimposed on the display screen 160 a depictedin FIG. 10. In this case, when the content viewer points and determinesthe arrow 164 by a cursor or the like through the input device 14 b, thedisplay-picture generation section 76 moves the viewpoint toward thedirection of the arrow A, if needed, as in the case of FIG. 9. At thesame time, the visual line is controlled to be directed toward thebuilding 146, as depicted on the left side in FIG. 10.

Specifically, the display-picture generation section 76 switches thedisplay target to the original picture 142 b photographed from the nextphotograph viewpoint 152 b in the advance direction, and then, decidesthe visual line by referring to tag data 162 b such that the region 154b including the tagged position is displayed. Accordingly, a pictureincluding the building 146 that is viewed from the moved viewpoint isdisplayed, as displayed in the display screen 160 b.

During the transition from the display screen 160 a to the displayscreen 160 b, the same building 146 is kept displayed. Therefore, alocation being displayed can be intuitively discerned, and further, thetarget building 146 can be checked in more detail. It is to be notedthat the method for selecting a landmark is not limited to that depictedin the drawing. For example, when a cursor which is moved by a vieweroverlaps a landmark provided with a tag, the color of the landmark maybe changed to indicate that the landmark is selectable.

Also in the present embodiment, once a picture, such as the displayscreen 140 b in FIG. 9, in which the building 146 is not included may bedisplayed, and then, display of a geometrical figure such as an arrowindicating a direction in which the building 146 is located may besuperimposed on the picture in the similar manner to that in FIG. 7.With the geometrical figure such as an arrow, an image of the building146 that is located in the direction indicated by the arrow may beclipped so that display of the clipped image is superimposed. In thiscase, when the content viewer changes the visual line to the directionindicated by the arrow through the input device 14 b, a visual fieldincluding the building 146, that is, the display screen 160 b can beeasily displayed. Also in this case, the target building 146 can beconfirmed with little labor and time while the building 146 and thecurrent position are not missed.

In addition, in map content, positional information regarding aphotograph viewpoint and map information can be combined to identify adirection in which the position of a tagged target is located withrespect to the photograph viewpoints. In a case where this technology isused, even after switching to an original picture in which a landmarksuch as the building 146 designated by a content viewer is not includedis performed, a movement direction for displaying an original pictureincluding the landmark can be indicated.

In the example in FIG. 10, for example, even if the photograph viewpoint152 b is not present in the advance direction and the building 146cannot be seen from the next photograph viewpoint (e.g. photographviewpoint 152 c), the direction of the building 146 can be indicated byan arrow or the visual line can be directed to the direction in thedisplay using the corresponding original picture. Accordingly, theviewer can bring the viewpoint closer to the building 146. After awhile, a close-up image of the building 146 can be represented by use ofan original picture photographed from a photograph viewpoint (e.g.photograph viewpoint 152 d) from which the building 146 can be seen.

Next, operation of the content reproduction device 20 that can beimplemented by the aforementioned configurations will be explained. FIG.11 is a flowchart depicting process procedures in which the contentreproduction device 20 according to the present embodiment generates adisplay picture from a plurality of whole-sky pictures while using tagdata. This flowchart is started in a state where a plurality of originalpictures, data on tags associated with the respective original pictures,and data indicating positional information regarding the photographviewpoints, are stored in the data storage section 72. In addition, itis assumed that content in which display is performed while switchingamong original pictures is performed according to a content viewer'soperation, is created.

First, the display-picture generation section 76 selects any one oforiginal pictures in the data storage section 72, and generates anddisplays an initial picture (S10). An original picture, a viewpoint, anda visual line which are used in this step are previously set for thecontent. Alternatively, a map may be displayed in advance, a contentviewer may designate a point thereon, and the original picture, theviewpoint, and the visual line may be determined on the basis of thedesignated point. Next, the display-viewpoint acquisition section 74changes the viewpoint and the visual line according to a contentviewer's operation (S12). As previously explained, the viewpoint and thevisual line may be directly operated by the content viewer or may bechanged as a result of information processing in an electronic game orthe like.

Next, the display-picture generation section 76 confirms whether or notoriginal picture switching is necessary because a photograph viewpointthat is different from that of the original picture used so far becomesoptimum (S14). When original picture switching is not necessary (N atS14), the display-picture generation section 76 does not performoriginal picture switching but generates a picture of a region locatedin the direction of the visual line, and outputs the picture to thedisplay device 16 b (S22). When original picture switching is necessary(Y at S14), the display-picture generation section 76 confirms whetheror not a tagged position is included in the previously displayed region,by referring to tag data associated with the original picture before theswitching (S16).

In a case where the tagged position is included in the visual field (Yat S16), the display-picture generation section 76 identifies a positionat which a tag for the same subject is provided, by referring to tagdata associated with the switched original picture. Then, thedisplay-picture generation section 76 decides the visual line such thatthe visual field includes the position (S18). In a case where a taggedposition is not included in the visual field before the switching (N atS16), the visual line is decided in accordance with a predeterminedrule. For example, the direction of the visual line before the switchingis kept. After deciding the visual line in 51 or S20, thedisplay-picture generation section 76 generates a picture of a regionlocated in the direction of the visual line by using the switchedoriginal picture, and outputs the generated picture to the displaydevice 16 b (S22).

When there is no necessity (e.g. a request to stop content viewing fromthe content viewer) to end the process, S12 to S22 are repeated. Whenthere is a necessity to end the process, the process is stopped (Y atS24). Accordingly, picture display using a plurality of originalpictures can be realized while an image which is displayed ismaintained, if needed, in a visual field corresponding to a contentviewer's operation. It is to be noted that S18 may be executed by anoperation performed on predetermined input means, as depicted in FIG. 8,but an explanation of such a case is omitted in this flowchart.

According to the present embodiment explained so far, a technology ofusing, as display targets, original pictures, such as whole-skypictures, photographed from a plurality of viewpoints, and switching anoriginal picture for use in display, thereby displaying a picture fromvarious viewpoints, is provided in such a way that tag data indicatingthe position at which a subject is to be displayed is prepared to beassociated with each of the original pictures. Then, a visual line iscontrolled such that, in a case where a tagged position is included in avisual field before original picture switching, a position indicated bya tag for the same subject is continuously included in the visual fieldafter the switching. Consequently, a main subject can be continuouslydisplayed before and after original picture switching.

As a result, a viewer can be prevented from missing a point to see ormissing the position of the viewer in the picture space when an originalpicture is switched. In addition, labor and time to search a widepicture space in order to catch a target that the viewer was viewingbefore the switching, can be saved. Not only when original pictureswitching is performed, but also when a content viewer operatespredetermined input means, the visual field is moved to a taggedposition so that the visual line can be easily directed toward a mainsubject. Consequently, while a whole-sky picture is used to give a highdegree of freedom to a viewpoint or a visual line, the visual line isproperly controlled at a necessary timing, whereby display of a picturecan be continued in a preferable visual field as a whole.

The present invention has been explained so far on the basis of theembodiments. The above embodiments exemplify the present invention. Aperson skilled in the art will understand that various modifications canbe made to a combination of the components and the processing processes,and that these modifications are also within the scope of the presentinvention.

REFERENCE SIGNS LIST

1 Content processing system, 10 Picture data output device, 12 Picturecapturing device, 14 a Input device, 16 a Display device, 18 Contentcreation device, 20 Content reproduction device, 23 CPU, 24 GPU, 26 Mainmemory, 32 Communication section, 34 Storage section, 36 Output section,38 Input section, 40 Recording-medium driving section, 50 Originalpicture acquisition section, 52 Tag data generation section, 54 Dataoutput section, 60 Data acquisition section, 62 Data storage section, 64Content generation section, 66 Data output section, 70 Data acquisitionsection, 72 Data storage section, 74 Display-viewpoint acquisitionsection, 76 Display-picture generation section, 78 Data output section.

INDUSTRIAL APPLICABILITY

As explained so far, the present invention can be used for variousdevices including game devices, image processing device, picture dataoutput devices, content creation devices, content reproduction devices,picture capturing devices, and head mounted displays, and for systemsincluding such devices.

1. A content reproduction device comprising: a data acquisition sectionthat acquires data on an original whole-sky picture for use in display,and tag data indicating a position of a particular subject in theoriginal picture; a display-viewpoint acquisition section that acquiresa display-time viewpoint and a display-time visual line with respect tothe original picture; a display-picture generation section thatgenerates, from the original picture, a display picture in a visualfield based on the display-time viewpoint and the display-time visualline; and a data output section that outputs data on the display pictureto a display device, wherein, in a state where a predetermined conditionis satisfied, the display-picture generation section performs processingfor including the position in the original picture indicated by the taginto the visual field, by referring to the tag data.
 2. The contentreproduction device according to claim 1, wherein the data acquisitionsection acquires a plurality of original pictures obtained byfull-azimuth photographing from different viewpoints, and positionalinformation regarding viewpoints when the photographing is performed,and, when switching the original picture for use in the displayaccording to a change of the display-time viewpoint, the display-picturegeneration section changes the display by controlling the post-switchingvisual line such that the position indicated by the tag for a samesubject is included within the visual field before and after theswitching.
 3. (canceled)
 4. The content reproduction device according toclaim 1, wherein the display-picture generation section changes thedisplay by controlling the visual line such that the position indicatedby the tag is included within the visual field when the viewer operatespredetermined input means.
 5. The content reproduction device accordingto claim 1, wherein the data acquisition section acquires a plurality oforiginal pictures obtained by full-azimuth photographing from differentviewpoints, and positional information regarding viewpoints when thephotographing is performed, and, when switching the original picture foruse in display according to a change of the display-time viewpoint, thedisplay-picture generation section displays a geometrical figure toindicate, in the switched original picture, a relative direction of theposition indicated by the tag for a same subject that is included withinthe visual field before the switching.
 6. The content reproductiondevice according to claim 1, wherein the data acquisition sectionacquires the original picture data which is a whole-sky video includinga plurality of frames, and the tag data associated with the respectiveframes.
 7. A picture data output device that outputs data on an originalwhole-sky picture for use in display, the picture data output devicecomprising: an original picture acquisition section that acquires, asdata on the original picture, a video including a plurality of framesobtained by full-azimuth photographing from different viewpoints; a tagdata generation section that generates tag data for display control, thetag data indicating a position of a particular subject in each of theframes; and a data output section that outputs data on the originalpicture and the tag data in association with each other. 8.-9.(canceled)
 10. A content creation device comprising: a data acquisitionsection that acquires data on a plurality of original pictures obtainedby full-azimuth photographing from different viewpoints, and data ontags indicating positions of a particular subject in the respectiveoriginal pictures; a content generation section that generates videodata using the original pictures by setting data on an original picturefor use in display, a timing of switching the original picture, and avisual field to be displayed, with respect to a time axis of thedisplay; and a data output section that outputs the video data, whereinthe content generation section controls a post-switching visual field atthe timing of switching the original picture such that the positionindicated by the tag for a same subject is included within the visualfield before and after switching.
 11. A content reproduction method by acontent reproduction device, comprising: acquiring data on an originalwhole-sky picture for use in display, and tag data indicating a positionof a particular subject in the original picture; acquiring adisplay-time viewpoint and a display-time visual line with respect tothe original picture; generating, from the original picture, a displaypicture in a visual field based on the display-time viewpoint and thedisplay-time visual line; and outputting data on the display picture toa display device, wherein the generating the display picture involvesperforming, in a state where a predetermined condition is satisfied,processing for including the position in the original picture indicatedby the tag into the visual field, by referring to the tag data.
 12. Apicture data output method by a picture data output device that outputsdata on an original whole-sky picture for use in display, the picturedata output method comprising: acquiring, as data on the originalpicture, a video including a plurality of frames obtained byfull-azimuth photographing from different viewpoints; generating tagdata for display control, tag data indicating a position of a particularsubject in each of the frames; and outputting data on the originalpicture and the tag data in association with each other.
 13. A contentcreation method by a content creation device, comprising: acquiring dataon a plurality of original pictures obtained by full-azimuthphotographing from different viewpoints, and data on tags indicatingpositions of a particular subject in the respective original pictures;generating video data using the original pictures by setting data on anoriginal picture for use in display, a timing of switching the originalpicture, and a visual field to be displayed, with respect to a time axisof the display; and outputting the video data, wherein the generatingthe video data involves controlling a post-switching visual field at thetiming of switching the original picture such that the positionindicated by the tag for a same subject is included within the visualfield before and after the switching.
 14. A non-transitory, computerreadable storage medium containing a computer program, which whenexecuted by a computer, causes the computer to carry out actions,comprising: acquiring data on an original whole-sky picture for use indisplay and tag data indicating a position of a particular subject inthe original picture; acquiring a display-time viewpoint and adisplay-time visual line with respect to the original picture;generating, from the original picture, a display picture in a visualfield based on the display-time viewpoint and the display-time visualline; and outputting data on the display picture to a display device,wherein generating the display picture involves performing, in a statewhere a predetermined condition is satisfied, processing for includingthe position in the original picture indicated by the tag into thevisual field, by referring to the tag data.
 15. A non-transitory,computer readable storage medium containing a computer program, whichwhen executed by a computer, causes the computer to carry out actions,comprising: a function of acquiring, as data on the original picture, avideo including a plurality of frames obtained by full-azimuthphotographing from different viewpoints; a function of generating tagdata for display control, the tag data indicating a position of aparticular subject in each of the frames; and a function of outputtingdata on the original picture and the tag data in association with eachother.
 16. A non-transitory, computer readable storage medium containinga computer program, which when executed by a computer, causes thecomputer to carry out actions, comprising: a function of acquiring dataon a plurality of original pictures obtained by full-azimuthphotographing from different viewpoints and data on tags indicatingpositions of a particular subject in the respective original pictures; afunction of generating video data using the original pictures by settingdata on an original picture for use in display, a timing of switchingthe original picture, and a visual filed to be displayed, with respectto a time axis of the display; and a function of outputting the videodata, wherein the function of generating video data involves controllinga post-switching visual field at the timing of switching the originalpicture such that the position indicated by the tag for a same subjectis included within the visual field before and after the switching. 17.The content reproduction device according to claim 6, wherein thedisplay-viewpoint acquisition section switches the original picture foruse in display to an original picture photographed from a differentviewpoint, according to movement of the subject corresponding to thetag, and the display-picture generation section controls the visual linesuch that a position of the subject indicated by the tag at a time pointof the switching is included in the visual field of the switchedoriginal picture.